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A classical approach for dealing with the multiple testing prob- 
lem is to restrict attention to procedures that control the familywise 
error rate (FWER), the probability of at least one false rejection. 
In many applications, one might be willing to tolerate more than 
one false rejection provided the number of such cases is controlled, 
thereby increasing the ability of the procedure to detect false null 
hypotheses. This suggests replacing control of the FWER by con- 
trolling the probability of k or more false rejections, which is called 
the fc-FWER. In this article, a unified approach is presented for de- 
riving the fc-FWER controlling procedures. We first generalize the 
well-known closure principle in the context of the FWER to the case 
of controlling the fc-FWER. Then, we discuss how to derive the k- 
FWER controlling stepwise (stepdown or stepup) procedures based 
on marginal p- values using this principle. We show that, under cer- 
tain conditions, generalized closed testing procedures can be reduced 
to stepwise procedures, and any stepwise procedure is equivalent to a 
generalized closed testing procedure. Finally, we generalize the well- 
known Hommel procedure in two directions, and show that any gener- 
alized Hommel procedure is equivalent to a generalized closed testing 
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procedure with the same critical values. 

1. Introduction. Consider the problem of simultaneously testing a fi- 
nite number of null hypotheses Hi (i = 1, • • • , n), using tests that are avail- 
able for these individual hypotheses. A traditional concern dealing with this 
problem is to control the familywise error rate (FWER), the probability of 
falsely rejecting at least one true null hypothesis. However, quite often, when 
the number n of hypotheses is large, control of FWER is so stringent that a 
few of the false null hypotheses are rejected. Therefore, the classical idea of 
controlling the FWER has been recently generalized to that of controlling 
the probability of k or more false rejections, which is called the /c-FWER. 
The rationale behind the /c-FWER is that, often in practice, one is willing to 
tolerate a few false rejections, so by controlling k or more false rejections the 
ability of a procedure to detect more false null hypotheses can potentially 
be improved. 

A number of methods controlling the A;-FWER have been recently sug- 
gested. Single-step and stepwise (stepdown and stepup) fc-FWER procedures 
in terms of only the marginal null distributions of the p-values under ar- 
bitrary dependence of the p-values are derived in Hommel and Hoffmann 
(1987), Lehmann and Romano (2005), and Romano and Shaikh (2006). 
When the p-values are positively dependent, Sarkar (2008a) developed al- 
ternative single-step and stepwise fc-FWER procedures utilizing A;th order 
joint null distributions of the p-values. When the p-values are independent, 
Guo and Romano (2007) provided more powerful single-step and stepdown 
/c-FWER procedures. In Korn et al. (2004), alternative permutation-based 



3 

procedures are proposed to control the /c-FWER approximately, which ac- 
count for the dependence structure of the individual test statistics or p- 
values. Their results were generalized in Romano and Wolf (2007). In van 
der Laan et al. (2004), alternative procedures controlling the A;-FWER are 
provided by augmenting single-step and stepwise FWER procedures. Fur- 
ther methods are discussed in Dudoit et al. (2004) and van der Laan et al. 
(2005). 

In contrast to the popular false discovery rate (FDR), it is easier to de- 
rive powerful fc-FWER controlling procedures in numerous settings. For ex- 
ample, suppose we are examining all pairwise comparisons in the one-way 
ANOVA model, in which the number of treatments is moderate. In this sit- 
uation, the assumption of positive regression dependence of the underlying 
test statistics is not satisfied (Yekutieli, 2008), so Benjamini and Hochberg 
(1995) 's procedure is not applicable (Benjamini and Yekutieli, 2001), but 
no other FDR controlling procedure is available dealing with this problem. 
An alternative choice is to control the A;-FWER, since we will be able to 
develop relatively easy powerful fc-FWER controlling procedures accounting 
for the special dependence structure of the individual test statistics. Hence, 
in many applications, the fc-FWER can be regarded as a good complement 
to the FWER and FDR. For further enunciation of fc-FWER criterion, see 
Hommel and Hoffmann (1987), Korn et al. (2004), Lehmann and Romano 
(2005), and van der Laan et al. (2004). In addition, based on the similar 
rationale to the A;-FWER, Sarkar (2008b) advocated the A;-FDR using the 
expected ratio of k or more false rejections to the total number of rejec- 
tions, which is a generalization of the FDR. Several procedures controlling 
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the fc-FDR have also been developed in Sarkar (2008b) and Sarkar and Guo 
(2008a, b). 

In this paper, we focus on the control of the fc-FWER. Instead of pursuing 
a piece-meal approach, a unified approach is provided for the construction of 
the fc-FWER controlling procedures based on marginal p-values. The main 
motivation comes from one particular paradigm of research on the EWER, 
where the well-known closure principle plays a fundamental role in the con- 
struction of the EWER controlling procedures. We believe that a generaliza- 
tion of the closure principle will play a similar key role in the construction 
of the /c-FWER controlling procedures. To begin with, we generalize the 
closure principle, and then derive several general results on the relationship 
between generalized closed testing procedures and stepdown, stepup, and 
generalized Hommel procedures. As an application, it is then shown that 
the existing procedures can be directly derived following the generalized 
closure principle, and they are equivalent to some generalized closed testing 
procedures. 

This paper is organized as follows. In Section 4.2, we set up the termi- 
nology. A generalization of the closure principle and several global tests are 
provided in Section 4.3. In Section 4.4, we discuss the relationship between 
generalized closed testing procedures and stepdown and stepup procedures. 
Several general results are obtained. In Section 4.5, we generalize the Hom- 
mel procedure and show that generalized Hommel procedures are equivalent 
to generalized closed testing procedures with the same critical values. In Sec- 
tion 4.6, we offer some concluding remarks. 
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2. Basic Setting. Consider the problem of testing simultaneously a 
family of n null hypotheses Hi, . . . , Hn- Suppose that the family satisfies the 
free combination condition of Holm (1979), that is, for any / C {1, . . . ,n}, 
there exists a distribution P € il, for which all Hi,i € / are true and all 
Hi,i ^ I are false, where is the set of all possible distributions of the data. 

Suppose V is the number of true null hypotheses falsely rejected. The 
generalized family wise error rate (fc-FWER) is defined to be the probability 
of at least k false rejections, where k is pre-specified with 1 < k < n. That 
is, 

(2.1) /c-FWER = P{V > k} . 

If A; = 1, A;-FWER is the usual familywise error rate (EWER). When testing 
Hi, . . . , Hn, we assume that the values Pi, . . . ,Pn are available, and the 
p-values associated with the true null hypotheses satisfy 

(2.2) P{Pi <u} <u for any u G (0, 1). 

Let the ordered p- values be denoted by P(i) < • • • < P(n)^ ^-^id the associated 
hypotheses by . . . ,-ff(n)- Suppose < • • • < a,„ be a non-decreasing 
sequence of critical values. 

There are two main avenues open for developing multiple testing proce- 
dures based on marginal p-values: stepup or stepdown. We generalize these 
procedures to accommodate control of the A;-EWER. A (generalized) stepup 
procedure based on the critical values a^, which is slightly different from 
the usual one, is described below. If P(„) < a„, then reject all null hypothe- 
ses; otherwise, reject hypotheses -ff(i), • • • , -ff(r)) where r > A; is the smallest 
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index satisfying 

(2.3) > an,-- - >-P(r+l) > Oir+l. 

If, for all r > k, P(j.^ > Or, then reject the first (A; — 1) most significant 
hypotheses. 

Similarly, a (generalized) stepdown procedure, which is slightly different 
from the usual one, is described below. If > Uk, reject the first {k — 
1) most significant hypotheses. Otherwise, reject hypotheses -ff(i), ■ • • , -ff{r) 
where r > k is the largest index satisfying 

Note that, if k = 1, the stepwise (stepup or stepdown) procedures described 
above are the same as the usual ones. 

Evidently, from the definition of the A:-FWER, one can always reject the 
(k — l) most significant hypotheses without violating control of the A;-FWER. 
This is the reason why we give a slightly different definitions of stepup and 
stepdown procedures, in which, the (A; — 1) most significant hypotheses are 
automatically rejected. An alternative choice is to let aj = a^, 1 < i < k, 
as in Hommel and Hoffmann (1987) and Lehmann and Romano (2005). 
For convenience of discussion, in the subsequent sections, all procedures 
including the closed testing procedures described in Section 4.3 and the 
generalized Hommel procedures defined in Section 4.5, are also supposed to 
reject automatically the (/c — 1) most significant hypotheses. 

3. Generalized Closure Principle. In this section, we generalize the 
well-known closure principle in the context of the FWER to the case of 
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controlling the /c-FWER. Similar to the usual closure principle, the value 
of the generalized closure principle is that the problem of controlling the 
fc-FWER is reduced to the problem of controlling the usual probability of 
the Type I error of single tests of intersection hypotheses. 

Let / C {1, . . . ,n} with |/| > k. Let Pi,i ^ I be the p-values associated 
with Hi,i € / and Pij < • • • < P\i\:i an ordered arrangement of these p- 
values. Let Hij, . . . ,H^f^.j be the corresponding null hypotheses. Let Hj = 
r)i(zjHi. Let Qfc./ < • • • < be a given set of critical values. We present 
the following local test based on marginal p-values, which is proposed in 
Sarkar (2008a), for testing the single hypothesis Hf. 

(3.1) Reject Hj if and only if Pjj < aj j for at least one j € {k, . . . , |/|}, 

which implies that an intersection hypothesis is declared significant if and 
only if at least k of the individual hypotheses is found significant. The local 
test is denoted by (/ : ak-.i, ■ ■ ■ ,a\ii-j) and the Type I error probability 
associated with the local test is given by P{u\^if.{Pj:i < aj:i}) when the 
intersection hypothesis Hj is true. If the Type I error probability is < a, 
we call the local test as a level a test. Now, consider the family of the local 
tests {(/ : ak:i, ■ ■ ■ , a^j^.j) : / C {1, . . . , n}, |/| > k}. We call this family 
to be symmetric if for any subsets / and J with |/| = |J| > k, we have 
Oiij = cei:j, for k < i < \I\. The notation implies that we use the same 
local test for testing different intersection null hypotheses Hj and Hj if the 
cardinalities of / and J are the same. A symmetric family of local tests is 
indeed characterized by a double-integer- indexed set of critical constants. 
Later, for simplicity, we use (/ : a^, |/|, . . . , a|j| |j|) to denote the local test 
(/ : ak:i, . . . , in the symmetric family. The following lemma plays an 



important role in the construction of symmetric local tests. 

Lemma 3.1. (Rdhmel and Streitberg, 1987; Falk, 1989; Lehmann and 
Romano, 2005) Suppose Pi,...,Pt are p-values in the sense that P{Pi < 
u} < u for all i and u in (0, 1). Let their ordered values he < • • • < P{t)- 
Let = /3o < /?! < • • • < /3m ^ 1 for some m < t. Then 

m 

P {Pil) < . . . , P(r,^) < Pm} < tY.iP^ " • 

i=l 

For any subset / C {1, . . . ,n} with |/| > k, consider the local test (/ : 
ak,\i\i ' ' ' )C>^|/|,|/|) of P^i- Let m = t = \I\ and (3i = aj.|/| if A: < i < |/| and 
otherwise 0. Then, by Lemma 3.1, the Type I error rate of the local test is 
less than or equal to 

|/| 

(3-2) \I\{ak^\i\/k+ ^ (ai,|/i - ai-i,|/|)A) • 

i=k+l 

Evidently, if the right side of (3.2) is bounded above by a, then the local 
test is a level a test under arbitrary dependence of j5- values. 
Let afc < • • • < On be given. Define 

(3.3) D,{k)= max ||/|^^^ + |/| V «"-|/|+. " «n-|/|+.-i 

Then by (6), the local test (/ : aa„„|/|+j/Di(A;), k < i < \I\) is a level a 
tests of Hj. Specially, suppose (j = k, . . . , \I\) is a constant. For the 

validity of (3.2) < a, one only needs a^j/i < ka/\I\. Thus the local test 
(I : ka/\I\, • ■ • , ka/\I\) is also a level a test of Hj. 

After obtaining the symmetric families of local tests, we now generalize 
the usual closure principle for controlling the fc-FWER. 
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Theorem 3.1. Let {Hi,i = l,...,n} be a finite family of hypotheses 
and k be pre-specified with 1 < k < n. For any I C {1, . . . , n} with \I\ > k, 
let Pi,i I be the p-values associated with Hi,i E / and Pij < • • • < P\i\:i 
an ordered arrangement of these p-values. Let Hj = f]j^jHj. Suppose a 
level a local test defined by (3.1) is available to test Hj for each L. Then, 
the (generalized) closed testing procedure, which rejects any hypothesis Hi if 
and only if Hj is rejected for all L satisfying i £ L and Pi > Pk j, controls 
the k-FWER at level a. 

Proof. Let Lq be the set of indices of true hypotheses. Assume |/o| > k 
or there is nothing to prove. Define the event 

A = {at least k true hypotheses are rejected}. 

The occurrence of event A imphes that there exists i > k such that the null 
hypothesis Hi-j^^ is rejected. From the description of the generalized closed 
testing procedure, Hi-j^ is rejected implies that -ff/g is rejected. Therefore, 

k-FWER = P{A) < P{Hio rejected} < a. 

□ 

Remark 3.1. If A; = 1, the generalized closed testing procedure de- 
scribed above is identical to the one proposed by Marcus et al. (1976). Under 
such closed testing procedure, -ff(i), . . . ,LL(^k-i) '^i^ always be rejected. 

Remark 3.2. Compared with the original closure principle, the gen- 
eralized closed testing procedure involves far fewer single tests for testing 
significance of Hi when k >2. One of the reasons is that, when testing Hj, 
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{k — 1) false rejections are allowed. If Pi is one of the first {k — 1) minimal 
p- values in Pj,j € /, the p- value Pi does not play any role in testing Hj. 

In what follows, the closed testing procedures considered are always built 
on symmetric families of local tests characterized by the critical constants 

4. Stepwise Procedure. In this section, we discuss how to apply the 
generalized closure principle enunciated in Theorem 3.1 to derive stepwise 
(stepup or stepdown) procedures with the fc-FWER controlling property. 

It is generally not easy to show directly that a specific stepwise procedure 
has the /c-FWER controlling property. However, our strategy is first to build 
a closed testing procedure based on a family of level a local tests, and then 
to prove that the specific stepwise procedure is equivalent to or dominated 
by the closed testing procedure. We now qualify equivalence or dominance 
of two procedures (Liu, 1996; Grechanovsky and Hochberg, 1999). Two pro- 
cedures A and B are called equivalent if they reject or accept the same 
individual hypotheses. Procedure A is said to dominate Procedure B ii A 
always rejects at least those individual hypotheses rejected by Procedure B. 
It is easy to see that, if Procedure B is shown to be equivalent to or dom- 
inated by Procedure A, which has the fe-FWER controlling property, then 
Procedure B also has the A;-FWER controlling property. We now focus on 
stepdown procedures. 

4.1. Stepdown Procedure. When the p- values are in any dependency struc- 
ture, Hommel and Hoffmann (1987) and Lehmann and Romano (2005) pro- 
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posed a stepdown procedure with the critical values aj defined by, 

(4.1) ai = ^—t"' i = k,---,n, 

n — I + k 

with the A;-FWER controlling property, which is a generalization of Holm's 
procedure (Holm, 1979). In this subsection, we provide a general result (The- 
orem 4.1) of deriving stepdown procedures through given closed testing pro- 
cedures. 

Theorem 4.1. Suppose k < i < \I\ and k < \I\ < n he given, 
is increasing in i and decreasing in \I\. The following statements are true. 

(i) The closed testing procedure with the critical values dominates the 
stepdown procedure with the critical values a}.^[n-i)+k^ k < i < n. 

(ii) Furthermore, i/ai,|/|, k < i < \I\ are constant for each given then 
these two procedures are equivalent. 

(Hi) If {I : aj |7|,/c < i < \I\) is a level a local test of Hj for each I C 
{l,...,n} with \I\ > k, then the stepdown procedure controls the k-FWER 
at level a. 

Proof, (i) We first show that, for any individual hypothesis with 
index (i), which corresponds to the ith minimal p- value if it is rejected 
by the stepdown procedure, it is also rejected by the closed testing procedure. 

If i < fc, is automatically rejected by these two procedures. We as- 
sume i > A;. If f/'i'j-) is rejected by the stepdown procedure, then Pj-jj < 
(^k,{n-j)+k^ for all A; < j < i. Consider any subset / C {l,...,n} with 
(i) G I, \I\ > k, and > Pkj- Suppose Pkj = -P(/). Then 

k <l <i and |/| < A; + (n - I). 
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Consequently, 

Pk:I = P(l) < afc,(n-0+fc ^ 

That is, ff/ is rejected. Then, from Theorem 3.1, H(j^ is rejected by the closed 
testing procedure. Therefore, the stepdown procedure with the critical values 
k < i < n is dominated by the closed testing procedure. 

(ii) We now show that, when |/|, k < i < \I\ are constant for each given 
if is rejected by the closed testing procedure, it is also rejected by 

the stepdown procedure. 

Let Ij = {{l),...,{k - l),{j),...,{n)}, k < j < i. Note that, (i) G 
Ij, \Ij\ = {n — j) + k > k, and Pk-.ij = P{j)- If -H^(i) is rejected by the closed 
testing procedure, then, from Theorem 3.1, each Hj. will also be rejected by 
the corresponding level a local test. Note that, k < i < \I\ are constant 
for each given |/|, then Hj. is rejected if and only if Pk-.ij < '^k,\ij\- That 
is, P(j) < af:^(^n-j)+kj k < j <: i- Hence, from the definition of generalized 
stepdown procedure, is rejected by the stepdown procedure with the 
critical values ak^(^n-i)+k- So, these two procedures are equivalent. 

(iii) Since (I : Qj |/|,A; < i < \I\) is a level a local test for each / C 
{l,...,n} with j/| > k, then, from Theorem 3.1, the closed testing pro- 
cedure controls fc-FWER at level a. Note that, the stepdown procedure is 
dominated by the closed testing procedure, so the stepdown procedure also 
controls the A;-FWER at level a. □ 

Theorem 4.1 shows that under certain conditions, a generalized closed 
testing procedure can be reduced to a corresponding stepdown procedure. 
As an application, we point out that Hommel and Hoffmann (1987) and 
Lehmann and Romano (2005) 's stepdown procedure can be derived by the 
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general result by choosing appropriate level a local tests. As illustrated in 
Section 3, the local test (/ : jj^a, • • • , |7|«) is a level a test of Hj for each 
/. Note that cti^\i\ = jfjCt) k < i < \I\ are constant for each given |/|. Thus, 
by Theorem 4.1, the closed testing procedure based on these local tests is 
equivalent to the stepdown procedure with the critical values defined by 
(4.1). 

We now focus on the converse of Theorem 4.1. In Theorem 4.2, we show 
that any stepdown procedure is equivalent to some closed testing procedure. 

Theorem 4.2. Let Ok an be given. Then, the stepdown proce- 

dure with the critical values ai, k < i < n is equivalent to a closed test- 
ing procedure with the critical values = a„_|/|+fci ^ ^ ^ ^ 1-^1 CLiT'd 
k < \I\ < n. 

Proof. Note that, Pi^\i\ = an-|7|+fe is increasing in i and decreasing in 
and for each given |/|, are constant. Then, from Theorem 4.1, the 
closed testing procedure with critical values /3j |/| = a„_|/|+fc is equivalent 
to the stepdown procedure with the critical values Pk,n-i+k = oti. □ 

Theorem 4.2 shows that the stepdown procedures can be viewed as a 
specific class of the closed testing procedures. 

4.2. Stepup Procedure. When the p-values are in any dependency struc- 
ture, Romano and Shaikh (2006b) obtained a stepup procedure with the 
critical values a- = aai/Di{k), where Ofc ^ ■ ■ ■ ^ are any non-decreasing 
constants and Di{k) is defined as in (3.3). In this subsection, we give a 
general result (Theorem 4.3) of deriving stepup procedures through given 
closed testing procedures. 
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Theorem 4.3. Suppose cti^\i\, k < i < \I\ and k < \I\ < n be given, 
|/| is increasing in i and decreasing in \I\, and a; is increasing in 

I for each given i > k. The following statements are true. 

(i) The closed testing procedure with the critical values a.t,|/| dominates the 

stepup procedure with the critical values a/; 

(a) Furthermore, if ai^(^n-i)+h k <l <n are constant for each given i > k, 
then these two procedures are equivalent. 

(Hi) If {I : ctj < i < \I\) is a level a local test of Hj for each I C 
{1, . . . ,n} with \I\ > k, then the stepup procedure controls the k-FWER at 
level a. 

Proof, (i) We show that, for any individual hypothesis H{^i), if it is 
rejected by the stepup procedure, it is also rejected by the closed testing 
procedure. 

If i < fc, is automatically rejected by these two procedures. So, we 
assume i > k.li Hf^^-j is rejected by the stepup procedure, then there exists 
j > i satisfying P^^) < aj^^(^n-j)+k- Consider any subset I C {1, . . . ,n} with 
(i) € /, |/| > k, and P(j) > Pkj. Let I = max{f' : Pii-j < P{j)}. Since j > i 
and {i) e I, I > k.Ul < \I\, then Pi+i-j > Py), so \I\ < {n-j) + l. Evidently, 
when I = \I\, it follows that |/| < (n — j) + /, too. Hence, 

Pl:I < P{j) < ak^{n-j)+k < Oti^(^n-j)+l < 

The third inequality in the chain above follows from the assumption that 
is increasing in Z, and the last inequality follows from the inequality 
\I\ < {n—j)+l. Consequently, Hj is rejected. Then, by Theorem 3.1, is 
rejected by the closed testing procedure. Hence, the stepup procedure with 
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the critical values ak^(n-i)+k is dominated by the closed testing procedure. 

(ii) We show that when a/,(n-j)+/) k < I < n are constant for each given 
j > k, if //(j) is rejected by the closed testing procedure, it is also rejected 
by the stepup procedure. 

Let I = {(1), . . . ,{k — 1), (i), . . . , (n)}, and note that, (i) € /, |/| = (n — 
i) + k > fc. If is rejected by the closed testing procedure, then by 
Theorem 3.1, Hj will be rejected by the corresponding level a local test. 
That is, there exists j > i satisfying Pjj < Oij^\i\- Since Pjj = P(^i^j_i^), 
Oij^m = aj^n-i+k, and ai^(^n~i')+h k < I < n are constant for each given 
i' > k, aj^n-i+k = afc,n-j-i+2fc- Hence, P(i+j-k) < ak^n-{i+j-k)+k- From the 
definition of generalized stepup procedure, //(j) is rejected by the stepup 
procedure. Consequently, these two procedures are equivalent. 

(iii) Since (/ : Qj |/|,A; < i < \I\) is a level a local test of Hj for each 
/ C {1, . . . ,n} with |/| > k, by Theorem 3.1, the closed testing procedure 
controls the fc-FWER at level a. Note that the stepup procedure is domi- 
nated by the closed testing procedure, so the stepup procedure also controls 
the A;-FWER at level a. □ 

Theorem 4.3 shows that under certain conditions, a generalized closed 
testing procedure can be reduced to a corresponding stepup procedure. As 
an application, we show that Romano and Shaikh (2006) 's stepup procedure 
can be derived by the general result by choosing appropriate level a local 
tests. Let a/t <•••<«« be given and define /9j |7| = a„_|/|_|_j. As illustrated 
in Section 3, (/ : aa„_|/|+fc/£'i(fc), • • • , aun/ Di{k)) is a level a local test 
of Hj for each /. Note that j3i^(^n-i)+l = oii^ k < I < n are constant for 
each given i. Then, based on these local tests, Romano and Shaikh (2006) 's 
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stepup procedure is derived by Theorem 4.3. 

We now focus on the converse of Theorem 4.3. In Theorem 4.4, we show 
that any stepup procedure is equivalent to some closed testing procedure. 

Theorem 4.4. Let Ok < ■ ■ ■ < ctn be given. Then, the stepup procedure 
with the critical values ai, k < i < n is equivalent to a closed testing proce- 
dure with the critical values = Q!n-|/|+j) k < i < \I\ and k < \I\ < n. 

Proof. Note that, = is increasing in i, decreasing in 

and (3i^n-i+i = cti are constant for each given i > k. Then, from Theorem 4.3, 
the closed testing procedure with critical values = an-\i\+k is equivalent 
to the stepup procedure with the critical values Pk,n-i+k = on, k < i < 
n. □ 

Theorem 4.4 shows that the stepup procedures can also be viewed as a 
specific class of the closed testing procedures. 

Remark 4.1. When the values are positively dependent in the sense of 
being multivariate totally positive of order two (MTP2) (Karlin and Rinott, 
1980), based on the generalized Simes' test of Sarkar (2008a), we can easily 
derive the stepdown and stepup procedures in Sarkar (2008a) by Theorems 
4.1 and 4.3. 

5. Generalized Hommel Procedure. In the context of the FWER, 
Hommel (1988) developed a well-known sequential procedure based on Simes 
(1986)' test, which is more powerful than Hochberg (1988)'s stepup proce- 
dure (Hommel, 1989). Hommel's procedure is described as follows: compute 
j = max{i € {1, . . . ,n} : P(^n-i+l) > T' ^ ~ 1; ■ • • !^}- If the maximum 
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does not exist, reject all Hi [i = otherwise reject all Hi with 

Pi ^ Oi/ji = 1, • • • ,n). In this section, we generalize Hommel's procedure 
in two directions. In one direction, we move from the critical values of Simes' 
test to any double- indexed critical values satisfying certain properties, and 
in another, we move from the control of the FWER to that of the A;-FWER. 

For convenience of discussion, a formal definition of a generalized Hommel 
procedure is first given as follows. 

Definition 5.1. Let k < i < \I\ and k < \I\ < n he given. Suppose 
|/| is increasing in i and decreasing in \I\. A generalized Hommel procedure 
with the critical values 0(i^\i\ is defined as follows: compute j = max{i G 
{k,...,n} : P^n-i+l) > fori = k,...,i}. If the maximum does not 

exist, reject all H(^i^ [i = k,...,n), otherwise reject all H(^i-^ with P(^i^ < 
akj {i = k, . . . ,n). In any case, the first (k — 1) most significant hypotheses 
. . . , i?(fc„i) are automatically rejected. 

For example, for the original Hommel procedure, /c = 1, and is taken 
to be aj^i/i = p^Q!. The following is the main result of this section. 

Theorem 5.1. Let aj |/|,A; < i < |/| and k < |/| < n he given, and 
suppose is increasing in i and decreasing in \I\. Then the following are 
true. 

(i) The closed testing procedure with the critical values ajj/i is equivalent to 
the generalized Hommel procedure with the same critical values, 
(a) If (/ : k < i < \I\) is a level a local test of Hj for each I C 

{1, • • • ,n} with \I\ > k, then the generalized Hommel procedure controls the 
k-FWER at level a. 



18 

Proof, (i) If i < k, is automatically rejected by these two proce- 
dures. So, we assume i > k. Let j = max{i' € {k, . . . ,n} : P{^n-i'+l) > 
ai^i', for I = k, . . . Suppose j does not exist. By Definition 5.1, the gen- 
eralized Hommel procedure rejects all Hi. We now show that the closed 
testing procedure rejects all Hi. Non-existence of j implies that for any i 
with k < i < n, there exists / with k < I < i satisfying P(^n_i^i) < ai^i. 
Consider any subset / C {1, . . . ,n} with |/| = i. Suppose Pij = P[v), then 
|/| < (n — /') + /. That is, /' < n — i + /. So, Pij < P(^n-i+i) ^ Hence, 
Hj is rejected. By Theorem 3.1, all hypotheses -ff(i), • • • ,-ff(n) are rejected 
by the closed testing procedure. 

We now consider the case that j exists. First, we show that, for any indi- 
vidual hypothesis -ff(j), if it is rejected by the generalized Hommel procedure, 
it is also rejected by the closed testing procedure. 

If is rejected by the generalized Hommel procedure, then < Oi^j- 
Consider any subset / C {1, . . . ,n} with (i) € /, |/| > k, and Pj-j^ > Pkj- If 
|/| < j, then 

Pk:I < P{i) < OlkJ < 

Consequently, Hj is rejected. If |/| > j, then from the definition of j, there 
exists / satisfying A; < / < |/| such that P(n-\i\+i) ^ - = max{i' > 

k : Pi'j < P^n-\l\+l)}- Note that {\I\-k + l) + {n-\I\+l) = n + l-k + 1 > n, 
which implies that the maximum exists. From the definition of io, we have 
P'io+i-.i > P{n-\i\+i)j so |/| <n — {n— \I\ + 1) +io. That is, / < i^. Therefore, 

Pio-.l < P(n-\I\+l) < "«,|J| < "jo,|/|- 
Hence, Hj is rejected. Consequently, is rejected by the closed testing 
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procedure. Therefore, the generalized Hommel procedure with the critical 
values aj.|7| is dominated by the closed testing procedure. 

Next, we show that, if //(j) is rejected by the closed testing procedure, it 
is also rejected by the generalized Hommel procedure. 

Let / = {(1), . . . , {k - 1), {n - j + k), {n - j + k + 1), . . . , (n)}. Note that 
l-^l = J ^ ^1 and Pi- J = < I < j- From the definition of j, we 

have P[n-j+i) > all A; < / < j. Thus, Hi will not be rejected by 

the local test (/ : Qjj, k < i < j). li H^^^ is rejected by the closed testing 
procedure, then from Definition 5.1, (i) ^ I. That is, k < i < n — j + k. 
Let /' = {(1), . . . , (/c - 1), (i), {n - j + k + 1),. . . , (n)}. Note that (i) G I', 
|/'| = j > k and Pkj' = P{i)- Then Hp is rejected following from Theorem 
3.1. It now follows that Pj-j) < a^j and -ff(j) is rejected by the generalized 
Hommel procedure. Hence, these two procedures are equivalent. 

(ii) Since (/ : |/|, k < i < \I\) is a level a local test for each / C 
{1, . . . ,n} with |/| > k, then, from Theorem 3.1, the closed testing proce- 
dure controls the /c-FWER at level a. Since the generalized Hommel proce- 
dure is equivalent to the closed testing procedure, the generalized Hommel 
procedure also controls the /c-FWER at level a. □ 

Theorem 5.1 shown that each generalized closed testing procedure is 
equivalent to a generalized Hommel procedure with the same critical val- 
ues. Since generalized Hommel procedures are much simpler than closed 
testing procedures, this result implies that we find a new shortcut, which 
can reduce generalized closed testing procedures to computationally simple 
procedures in applications. 



20 

Remark 5.1. Professor Hommel brought to our attention, while com- 
menting on an earlier draft of the paper, that a version similar to Theorem 
5.1 for k = I appears in the paper by Bernhard et al. (2004). Their result is 
stated without proof. 

Finally, we provide an intuitive interpretation for the generalized Hommel 
procedures. For each generalized Hommel procedure with the critical values 
aj^|/|, the value j = max{i € {k, . . . ,n} : P[n-i+l) > for / = /c, . . . ,i} 
can be viewed as an estimate of the number of true null hypotheses. For ex- 
ample, suppose aj |/| = |^a. Let j = max |i G {A;, • • • , n} : P(„_„j+fc) > f a|, 
then j can be expressed as n—j + k, where j = min |j G {k, . . . , n} : > 
„_j-l-fc Q;|- Note that j — 1 is the number of rejected hypotheses by using the 
stepdown procedure of Hommel and Hoffmann (1987) and Lehmann and 
Romano (2005), and (k — 1) is the maximal number of false rejections that 
one is willing to tolerate in this procedure. Thus we use n — {j — l) + (k — l) = 
n — j + k = j as an estimate of the number of true null hypotheses in the 
family of hypotheses Hi, . . . , Hence, each generalized Hommel procedure 
with the critical values can be interpreted as a two-stage procedure, in 
which, first estimate the number of true null hypotheses by using j, and then 
based on the estimate j, establish a single-step procedure with the critical 
value j . 

6. Concluding Remarks. The original closure principle was formu- 
lated by Marcus et al. (1976) in the context of the FWER and has since 
been a powerful tool for deriving multiple testing procedures controlling the 
FWER. In fact, almost all FWER controlling procedures are either derived 
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using this principle or can be rewritten as associated closed testing proce- 
dures. The only disadvantage is that closed testing procedures are compu- 
tationally complex. 

In this paper, we have generalized the closure principle for the /c-FWER. 
In the same vein as the usual closure principle, the value of generalized clo- 
sure principle is that, instead of simultaneously testing multiple hypotheses, 
one only needs to do a number of single tests of intersection null hypotheses. 
The construction of single tests based on marginal p- values is relatively easy. 
The reason is that, as all p-values associated with true null hypotheses are 
marginally stochastically dominated by uniformly distributed random vari- 
ables, some powerful probability inequalities on uniformly distributed ran- 
dom variables are available such as Bonferroni inequality, Simes' inequality 
and generalized Simes' inequality, which are useful in building single tests. 
See Sarkar (2008a). 

The generalized closed testing procedures are also computationally com- 
plex. We have discussed how to reduce generalized closed testing procedures 
to simple stepwise procedures, and showed that, under certain conditions, 
a generalized closed testing procedure can be formulated as a stepwise pro- 
cedure. We have generalized the well-known Hommel procedure, and shown 
that each generalized closed testing procedure is equivalent to a simple gen- 
eralized Hommel procedure with the same critical values. 

We need to point out that, in this paper, we only discussed how to derive 
the /c-FWER controlling procedures based on marginal p-values using the 
generalized closure principle. A future research is to use this principle for de- 
veloping some resampling-based methods such as those described in Dudoit 
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et al. (2004), van der Laan et al. (2005), and Romano and Wolf (2007) that 
take into account the dependence structure of the underlying test statistics. 

We also need to note that the generalized closure principle is derived for 
the families of non-hierarchical null hypotheses. In many cases, we need to 
test families of hierarchical null hypotheses simultaneously. An interesting 
future research is to modify the generalized closure principle for families of 
hierarchical null hypotheses extending the work of Hommel (1986) on the 
FWER to the A;-FWER. 
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